Search for: All records

Reinforcement Learning from Human Feedback (RLHF) has become the predominant approach for language model (LM) alignment. At its core, RLHF uses a margin-based loss for preference optimization, specifying ideal LM behavior only by the difference between preferred and dispreferred responses. In this paper, we identify a common pitfall of margin-based methods -- the under-specification of ideal LM behavior on preferred and dispreferred responses individually, which leads to two unintended consequences as the margin increases: (1) The probability of dispreferred (e.g., unsafe) responses may increase, resulting in potential safety alignment failures. (2) The probability of preferred responses may decrease, even when those responses are ideal. We demystify the reasons behind these problematic behaviors: margin-based losses couple the change in the preferred probability to the gradient of the dispreferred one, and vice versa, often preventing the preferred probability from increasing while the dispreferred one decreases, and thus causing a synchronized increase or decrease in both probabilities. We term this effect, inherent in margin-based objectives, gradient entanglement. Formally, we derive conditions for general margin-based alignment objectives under which gradient entanglement becomes concerning: the inner product of the gradients of preferred and dispreferred log-probabilities is large relative to the individual gradient norms. We theoretically investigate why such inner products can be large when aligning language models and empirically validate our findings. Empirical implications of our framework extend to explaining important differences in the training dynamics of various preference optimization algorithms, and suggesting potential algorithm designs to mitigate the under-specification issue of margin-based methods and thereby improving language model alignment. 

Abstract. The hydrology of thawing permafrost affects the fate of the vast amount of permafrost carbon due to its controls on waterlogging, redox status, and transport. However, regional mapping of soil water storage in the soil layer that experiences the annual freeze-thaw cycle above permafrost, known as the active layer, remains a formidable challenge over remote arctic regions. This study shows that Interferometric Synthetic Aperture Radar (InSAR) observations can be used to estimate the amount of soil water originating from the active layer seasonal thaw. Our ALOS InSAR results, validated by in situ observations, show that the thickness of the soil water that experiences the annual freeze-thaw cycle ranges from 0 to 75 cm in a 60-by-100-km area near the Toolik Field Station on the North Slope of Alaska. Notably, the spatial distribution of the soil water correlates with surface topography and land vegetation cover types. We found that pixel-mismatching of the topographic map and radar images is the primary error source in the Toolik ALOS InSAR data. The amount of pixel misregistration, the local slope, and the InSAR perpendicular baseline influence the observed errors in InSAR Line-Of-Sight (LOS) distance measurements non-linearly. For most of the study area with a percent slope of less than 5%, the LOS error from pixel misregistration is less than 1 cm, translating to less than 14 cm of error in the soil water estimates. 

A paired alkaline electrolyzer with non-noble metal catalysts was developed, demonstrating higher performances of furfural oxidation on NiFe/Ni foam at the anode and hydrogen evolution on Co/MXene at the cathode under practical current densities. 

An important outstanding challenge that must be overcome in order to fully utilize the XY surface code for correcting biased Pauli noise is the phenomenon of fragile temporal boundaries that arises during the standard logical state-preparation and measurement protocols. To address this challenge we propose a new logical state-preparation protocol based on locally entangling qubits into small Greenberger-Horne-Zeilinger-like states prior to making the stabilizer measurements that place them in the XY-code state. We prove that in this new procedure $O\left(\sqrt{n}\right)$high-rate errors along a single lattice boundary can cause a logical failure, leading to an almost quadratic reduction in the number of fault configurations compared to the standard state-preparation approach. Moreover, the code becomes equivalent to a repetition code for high-rate errors, guaranteeing a 50% code-capacity threshold during state preparation for infinitely biased noise. With a simple matching decoder we confirm that our preparation protocol outperforms the standard protocol in terms of both threshold and logical error rate in the fault-tolerant regime where measurements are unreliable and at experimentally realistic biases. We also discuss how our state-preparation protocol can be inverted for similar fragile-boundary-mitigated logical-state measurement. Published by the American Physical Society2024 

Dueling bandits is a prominent framework for decision-making involving preferential feedback, a valuable feature that fits various applications involving human interaction, such as ranking, information retrieval, and recommendation systems. While substantial efforts have been made to minimize the cumulative regret in dueling bandits, a notable gap in the current research is the absence of regret bounds that account for the inherent uncertainty in pairwise comparisons between the dueling arms. Intuitively, greater uncertainty suggests a higher level of difficulty in the problem. To bridge this gap, this paper studies the problem of contextual dueling bandits, where the binary comparison of dueling arms is generated from a generalized linear model (GLM). We propose a new SupLinUCB-type algorithm that enjoys computational efficiency and a variance-aware regret bound $$\tilde O\big(d\sqrt{\sum_{t=1}^T\sigma_t^2} + d\big)$$, where $$\sigma_t$$ is the variance of the pairwise comparison in round $$t$$, $$d$$ is the dimension of the context vectors, and $$T$$ is the time horizon. Our regret bound naturally aligns with the intuitive expectation in scenarios where the comparison is deterministic, the algorithm only suffers from an $$\tilde O(d)$$ regret. We perform empirical experiments on synthetic data to confirm the advantage of our method over previous variance-agnostic algorithms. 

Abstract Over the last ten years, there has been considerable progress in using digital behavioral phenotypes, captured passively and continuously from smartphones and wearable devices, to infer depressive mood. However, most digital phenotype studies suffer from poor replicability, often fail to detect clinically relevant events, and use measures of depression that are not validated or suitable for collecting large and longitudinal data. Here, we report high-quality longitudinal validated assessments of depressive mood from computerized adaptive testing paired with continuous digital assessments of behavior from smartphone sensors for up to 40 weeks on 183 individuals experiencing mild to severe symptoms of depression. We apply a combination of cubic spline interpolation and idiographic models to generate individualized predictions of future mood from the digital behavioral phenotypes, achieving high prediction accuracy of depression severity up to three weeks in advance (R²≥ 80%) and a 65.7% reduction in the prediction error over a baseline model which predicts future mood based on past depression severity alone. Finally, our study verified the feasibility of obtaining high-quality longitudinal assessments of mood from a clinical population and predicting symptom severity weeks in advance using passively collected digital behavioral data. Our results indicate the possibility of expanding the repertoire of patient-specific behavioral measures to enable future psychiatric research.